Bug Proofing Visual Basic: A Guide to Error Handling and Prevention
(Publisher: John Wiley & Sons, Inc.)
Author(s): Rod Stephens
ISBN: 0471323519
Publication Date: 11/01/98

Do not stop testing simply because you are sick of it, you have used up your test cases, or because you have reached the scheduled end date for testing. Remember, the longer a bug stays in the program, the harder it will be to fix. Fixing bugs is easier during testing than it is after you have put the finishing touches on the application. It is also less embarrassing than letting your customers find bugs and telling all their friends and business associates that you write buggy software.

Stop testing when testing is no longer profitable; in other words, when it no longer finds bugs.

Estimate the Number of Bugs

If you can accurately estimate the number of bugs remaining, and if you can predict the speed with which you will find and fix the bugs, you can guess how long you will need to continue testing. Unfortunately, neither of these estimates is easy.

Estimating the number of bugs remaining is hard. Often in programming classes a student has asked me for help, saying, “I just have one more bug.” The student assumes there is only one bug due to lack of experience. Really there are usually several bugs; the student just knows about one.

There are several ways you can estimate the number of bugs remaining in a program. One method is to guess based on previous experience or industry averages. If this project team produced three errors per hundred lines of code in the past, and if this project is similar to the previous ones, then the current project will probably contain about three errors per hundred lines of code.

Another method is to examine a large piece of code extremely closely and count the number of bugs it contains. You then assume the rest of the project has a similar number of bugs per line of code. For this method to be useful, you need to be relatively certain you have found most of the bugs in the test code. Pieces of the code must also have been written by the developers in proportion to their contributions to the project as a whole. If one programmer wrote 10 percent of the project’s code, 10 percent of the sample should come from that programmer. Finally, the sample code should be representative of the project as a whole. If only 5 percent of the project is mail processing code, the sample must not contain 95 percent mail processing code.

One intriguing technique for bug estimation is to seed the code with artificial bugs. After a certain amount of testing, you compute the fraction of the seed bugs that have been found. If the testers have found half of the seed bugs, they have hopefully found roughly half of the other bugs, too. Of course, if the seed bugs do not resemble the real bugs, the estimate will be inaccurate. If the code contains a lot of bugs involving Internet communications but the seed bugs contain none, those bugs will not be counted.

These techniques cannot guarantee accurate predictions, but they can give you a rough estimate with which to work. If you estimate the project contains around 470 bugs and you have found only 125, you need to continue testing. If your current testing methods are not finding bugs, you should design some new tests. You should not rest with an estimated 345 bugs outstanding until you have made an extraordinary effort to find them. Chances are they are there waiting for your customers to find them.

Estimate Bug Finding Time

Once you have a guess for the number of bugs remaining, you need to estimate the speed with which the testers can find the bugs. This is not easy, either. Some bugs are harder to find than others. As the number of bugs remaining decreases, it will probably become harder to find new bugs. The bugs that remain will also have been in the code the longest so they will be the hardest to fix and the most likely to cause new bugs.

Figure 14.6  A graph of bugs found per week.

If the tests do not expose all of the possible kinds of bugs, you may not know others exist until late in the testing process. For example, if none of the tests you used to predict the bug count exercises the electronic mail interface, lots of bugs may be hidden in that part of the code.

One simple method for estimating bug catching time is to graph the number of bugs found per unit time in the past. The graph may reveal an obvious trend. In Figure 14.6, the number of bugs found per week wobbles around a bit, drops quickly, and then starts to level out. If this trend continues, the team will not find many bugs in the next few weeks. At this point, the effectiveness of the tests is declining. If you have not found a substantial fraction of the bugs estimated, you should design a new set of tests to locate the hidden bugs.

Test User Interfaces from the Top Down

Visual Basic makes top-down development easy. Once you have a solid design, you can quickly put together the application’s main forms. You can then incrementally add controls, menus, and code behind them. The system develops in a top-down fashion. The high-level objects like forms are created first and the low-level coding details are added later.

As the project develops, you can test it in a top-down fashion. When you add a command button to a form, test it. Initially write a dummy event handler that just presents a message box saying it is there. As you fill in the details, test them.

In order to test some routines, you may need to write other dummy routines.If a subroutine calls three functions, you will need to write dummies for each of them. These dummies must take reasonable inputs and produce correct outputs so you can test the calling routine. It does not matter how the dummies produce their results, however. For example, they may look up results that have been computed by hand and stored in a text file or database.

Because of the need for extensive test cases and dummy routines that can simulate them, it is hard to analyze elaborate code subsystems using top-down testing. On the other hand, testing user interfaces is relatively easy. You can create the forms, buttons, and menus. Add the code that displays different kinds of forms and dialogs. Provide simple dummies for code that does not deal directly with the user interface.

Using this technique, you can test the user interface code quickly and verify that the program behaves properly on the surface. You can also show this interface to your customers early in the project and get feedback before you have written a huge amount of code.

Test Code from the Bottom Up

Intricate code systems are hard to test using a top-down approach. A bottom-up strategy can make testing them much easier.

First, test the routines that do not call any other routines. Once they are thoroughly tested, test routines that call only routines that have been tested. Continue the process, testing more complex routines using only tested routines, until you have tested all of the code.

At this point, integrate the code that was tested from the bottom up with the user interface that was tested from the top down. Now you can test the complete application. If you have properly tested the user interface code and the more complex application code, you should not find many bugs at this stage.